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AB STRACT 



The Michigan English Study of Structure for 
Curriculum Evaluation (MESSAGE) training session involved 40 leaders 
of English curriculum study in a 7-day intensive program. They 
studied objective evaluation in terms of behavior, instruction, and 
institution; approaches to the English curriculum through 
instructional objectives that include evaluation procedures; and 
implementation of training by working with local curriculum groups. 
Through participation in the program, curriculum leaders should be 
able to develop criteria for evaluating English programs, English 
materials, and individual student progress; gather information about 
local programs; and exercise leadership in planning programs, writing 
objectives, and evaluating materials and programs. Local schools 
should be able to (1) find nearby curriculum leaders to help with 
basic planning, (2) utilize program participants for training other 
staff members, and (3) evaluate present curriculum procedures and 
prepare changes based on rational methods of innovation. (Appendices 
include the EPIC Evaluation Center* s objectives for the Project 
MESSAGE program, a list of participants, suggestions for writing 
auestions, and feedback questionnaires.) (Author/LH) 
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FOREWORD 



The MESSAGE (Michigan English Study of Structure for Curricu- 
lum Evaluation) training session involved forty curriculum 
leaders in English, selected from the state at large, who 
have major responsibility for working with school staffs in 
the development of curricula. The seven-day training pro- 
gram, described in this report, was used to introduce the 
principles of systematic evaluation, to provide practice in 
the writing of relevant behavioral instructional objectives, 
and to prepare participants to apply long-range systematic 
evaluation procedures to the preparation of English programs 
for schools. 

The Michigan Council of Teachers of English sponsored MESSAGE? 
the U. S. Office of Education funded it? the Michigan Depart- 
ment of Education and administrators from throughout the 
state supported it. All of these agencies may expect that 
MESSAGE participants will provide state leadership in the 
areas of research, curriculum planning, and critical 
evaluation . 
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SUMMARY 
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MICHIGAN ENGLISH STUDY OF STRUCTURE FOR CURRICULUM 

EVALUATION 

(MESSAGE) 



The MESSAGE training session was a seven-day intensive pro-., 
gram for Michigan curriculum leaders in English. Forty par- 
ticipants from over the state , who had been identified as 
leaders or potential leaders of curriculum studies in 
English, were introduced to a three-dimensional model for 
evaluation. On this basis they studied objective evaluation 
in the context of the variables of behavior, instruction, 
and institution. Ways in which the English curriculum could 
be approached through the building of instructional objec- 
tives that would include evaluative procedures, and implemen- 
tation of the training through work with local curriculum 
groups. The emphasis throughout was on long-range evalu- 
ation of programs and curriculum building through a program 
of feedback of information about curricula. 

As a result of participation of staff members in the 
program, schools in the State should be able to: 

1. Find curriculum leaders in their own or nearby sys- 
tems to help with the basic planning for curriculum change. 

2. Utilize the training of certain staff members who 
have already been trained through Project English centers, 
summer institutes, or other formal or individualized pro- 
grams. With their backgrounds in subject-matter areas, 
these leaders will be able to lead the schools in combining 
techniques of evaluation and curriculum construction. 

3. Evaluate present curriculum patterns and base any 
proposed change on rational procedures of innovation. 

As a result of participating in this training program, 
curriculum leaders should be able to: 

1. Develop criteria for evaluating significant factors 
in English programs. 

2. Select, develop, and use a variety of methods for 
evaluating different English materials for comparative 
effectiveness . 
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3. Use a variety of resources for gathering infor- 
mation about local programs. 

4. Discuss and develop a variety of models for evalu- 
ating individual progress. 

5. Exert leadership in their own curriculum areas in 
the planning of programs, the writing of instructional ob- 
jectives, the selection of procedures to evaluate materials, 
and the evaluation of the programs. 






j 

1 

INTRODUCTION i 



The idea of the MESSAGE training program had its in-, 
ception in the spring of 1967 . The president-elect of the 
Michigan Council of Teachers of English, Dr. Vi Marie Taylor, 
began to talk with members of the executive board of the 
Council about ways in which the organization could help 
Michigan schools and teachers to face some of the problems 
in building sound English curricula for the schools of the 
state. Teachers in the schools were asking questions which 
apparently were going unanswered or being answered too quick- 
ly and without proper study, such as: Where should we start 

in selecting materials? Should we be adopting some of the 
suggested materials being developed by the Project English 
centers (which were then entering their final phases)? What 
should we do about initiating innovations or putting into 
practice some of the techniques being tried in various 
schools of the state? These questions seemed to be surface 
manifestations of deeper problems, namely: how do depart- 

ments of English determine their goals; how competently do 
they judge their achievements; how scientific is their 
measurement of progress toward the goals and objectives they 
have set? Apparently, many systems were looking for people 
to help them to find answers to these questions; the uni- 
versities were frequently asked to send help, or to supply 
short-term answers, but many systems were still looking for 
answers to questions that were symptomatic of problems that 
were not being reached. 

In the fall of 1967, as president of the Michigan 
Council, Dr. Taylor brought to the Executive Board of the 
Council a plan for a training session which would help 
school systems to find among their own people the personnel 
that could be developed to help to plan attacks on the basic 
problem. With the approval of the Executive Board, she 
wrote a proposal for the Michigan English Study of Structure 
for Curriculum Evaluation (MESSAGE) and secured a grant from 
the United States Office of Education, Research and Training 
Division, for the training of curriculum leaders in the de- 
sign and writing of objectives for the teaching of English. 
Cooperation was pledged by the National Council of Teachers 
of English and by the Michigan Department of Education. 

With the final approval of the grant, the training 
session was scheduled for the Boyne Mountain Lodge, Boyne 
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Falls, Michigan, April 8 - 24, 1969. The seven days im- 
mediately preceded the annual conference of the Michigan 
Council of Teachers of English at the same location. In 
this isolated location, the participants, consultants, and 
staff would work, eat, and play together for the period of 
the training session. They would also be able to end their 
training program by making immediate reports of the program 
to the members attending the annual conference. 



OBJECTIVES 



The objectives of the training program, as announced in 
the preliminary releases to the members of the Council, to 
administrators, and through news media of the Department of 
Education, suggested ways in which school systems might bene- 
fit from participation in the MESSAGE project: These general 

objectives were stated as follows: 

I. As a result of the participation of staff members 
in the program, schools in the state will be able to 

1. Find curriculum leaders in their own or nearby 
systems to help with the basic planning for curriculum 
change . 

2. Utilize the training of certain staff members 
who have already been trained through Project English 
centers, summer institutes, or other formal or indi- 
vidualized programs. With their backgrounds in subject- 
matter areas, these leaders will be able to lead the 
schools in combining techniques of evaluation and cur- 
riculum construction. 

3. Evaluate present curriculum patterns and base 
any proposed change on rational procedures of 
innovation. 

II. As a result of participating in this institute, cur- j 

riculum leaders will be able to ? 

1. Develop criteria for evaluating significant I 

factors in English programs. i 

2. Select, develop, and use a variety of methods 
for evaluating different English materials for com- 
parative effectiveness. 

3*-' Use a variety of resources for gathering infor- 
mation about local programs. 

4. Discuss and develop a variety of models for 
evaluating individual progress. 
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5. Exert leadership in their own curriculum areas 
in the planning of programs, the writing of instruction- 
al objectives, the selection of procedures to evaluate 
materials, and the evaluation of the programs. 

Specific behavioral objectives developed by the staff 
for the training session are included in Appendix I in the 
section outline for "Skill Development in Evaluation for 
Teachers of English." 



METHODS 



PRELIMINARY ANNOUNCEMENTS AND PARTICIPANT SELECTION 

Several means of information dissemination were em- 
ployed to notify potential participants about the program. 

The complete mailing list of the Council, consisting of over 
2200 teachers, administrators, and curriculum leaders in the 
field of English, as well as college and university person- 
nel, was used for the distribution of a special four-page 
edition of the Michigan English Teacher, which contained a 
preliminary application form- The mailing list of the State 
Department of Education, which lists over 1300 intermediate 
and system superintendents and principals, was used to send 
announcements and nomination forms. The State Department of 
Education also included announcements in its regular organs, 
which are sent to all administrators in the state . 

From these sources, more than 170 preliminary appli- 
cations were returned. These people were supplied with the 
longer application form, further information about require- 
ments, and the reminder that each application must be ac- 
companied by a statement from an administrator that the 
trainee would be utilized by the school system to lead in 
curriculum evaluation studies. More than one hundred com- 
pleted applications were received. Applications were read 
by four different readers, and selections were made on the 
basis of leadership potential? backgrounds in English, cur- 
riculum, and evaluation? geographic distribution? and availa- 
bility for service to schools other than their own. In ad- 
dition to the 40 participants selected for funding, four 
more were permitted to attend at their own or their school's 
expense. Every first choice appointee accepted the appoint- 
ment and attended the full session. 

Participants included: 

one consultant for 28 local school districts 

eight K-12 supervisors, coordinators, department heads, or 
curriculum directors 

four 7-12 supervisors, coordinators, department heads, or 
curriculum directors 



ten 



department heads or 
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9-12 supervisors, coordinators, 
curriculum directors 



four 10-12 supervisors, coordinators, department heads, or 
curriculum directors 



one 7-9 coordinator 



eleven chairmen of curriculum committees of local districts 
one university methods teacher (observer) 
one college teacher-consultant 

one elementary teacher, member of English curriculum com- 
mittee (observer) 

two secondary classroom teachers, members of English cur- 
riculum committees (observers) 

Administrators offered assurance that participants would 
be assigned a major role in the areas of planning and evalu- 
ation and that they would be encouraged to assist nearby 
schools in evaluation of programs and implementation of new 
ideas for the improvement of the teaching of English in the 
state o 



Names, addresses, and positions of participants are 
given in Appendix II. 



PROGRAM PLANNING SESSIONS 



Acting for the Executive Board of the Michigan Council 
of Teachers of English, a committee met in January to es- 
tablish preliminary guidelines and plan staff. This com- 
mittee was composed of Dr. Stephen Dunning, 1968-69 presi- 
dent of the Council; Miss Mildred Webster, executive 
secretary of the Council; Mr. Dwight Smith, English special- 
ist of the Michigan Department of Education; and Dr. Vi 
Marie Taylor, director of the program. 

This group set February 14-15 for a planning session 
and decided to invite Dr. Robert L. Hammond, Ohio State Uni- 
versity, and !;Dr. Kellogg Hunt, Florida State University, to 
the planning meeting. The committee learned that the repre- 
sentative of the National Council of Teachers of English at 
the planning session would be Mr. William Scannell, of the 
national office. At this meeting also, the dates April 18- 
24 were confirmed and the meeting place at Boyne Mountain 
Lodge, Boyne Falls, Michigan, was established. The time and 
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date were planned to precede the annual meeting of the 
Michigan Council of Teachers of English, scheduled for 
April” 25-27 . 

On February 14-15, specific elements of the program 
were developed in the planning session. The division of 
time among the various parts of the program was discussed, 
and approximately equal time segments were allocated to 
general evaluation procedures and the writing of objectives, 
on the one hand, and to the application of these techniques 
to English programs. Dr. Hammond presented in some detail 
the EPIC model for curriculum evaluation, which the project 
director had suggested as a structure for the training 
session. The English specialists present at the planning 
session agreed that the EPIC model could be used for the ex- 
amination of English curricula in the context of dimensions 
of behavior, instruction, and institution. It was decided 
to extend an invitation to Dr. R. J. Armstrong to bring a 
team from the EPIC Center, Tucson, to present the model to 
trainees and to lead the general evaluation segment. 

It was decided that Dr. Hunt would use a day and a half 
of the training session to apply the techniques of the pro- 
gram directly to programs in composition, syntax, and read-^ 
ing. Dr. Dunning would use one session of the training pro- 
gram for a demonstration of the teaching of poetry as a way 
of arriving inductively at the construction of behavioral 
goals. The methods employed by Dr. Alan C. Purves, Uni- 
versity of Illinois, were suggested for inclusion in the 
program as an additional objective instrument for measure- 
ment of English programs, and Dr. Purves was invited to par- 
ticipate in the training session. 
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PROGRAM 



General Evaluation Segment 

Throughout the training session, emphasis was placed on 
long-range evaluation techniques and their application to the 
design of English programs. For this purpose, the staff in- 
cluded instructors capable of covering this material from 
varied approaches. Dr. Robert L. Hammond set the tone for 
the entire session in his opening remarks. He enumerated 
problems faced by all curriculum designers, discussed present 
steps and processes available to correct these problems, and 
urged the development of evaluation procedures designed to 
utilize the capabilities of technological advances. In 
stressing the need for long-range evaluation programs, he 
presented four specific components of long-range evaluation. 
These four components of a program of long-range evaluation, 
used as a basis for the total training session, are as 
follows : 



1. Program identification 

2 . Collection of information about the program 

3. Organization of information about the program 

4. Analysis and reporting of the findings to the 
decision makers. 

Dr. Hammond also suggested desirable attributes for a system 
of evaluation that would operate on a long-range basis. 

Such a program would provide: 

1. An evaluation system that monitors the total in- 
structional program — that goes on year after year and pro- 
vides information to the teacher with respect to changes in 
the program. 

2 . An evaluation system that provides information as 
to what new programs might be selected. 

3. An evaluation program that monitors new programs 
and allows modification through evaluation of the program, 
not the students. 

4. An evaluation system that gives information about 
the year-long program. 

Dr. Robert Armstrong, director of EPIC, and his staff, 
introduced the three-dimensional EPIC model for the 
structure of evaluation and demonstrated how the structure 



provides for the identification of variables in the cate- 
gories of behavior, instruction, and institution. (The 
scheme is shown in detail and in diagram form in Appendix 
I, "Skill Development in Evaluation for Teachers of English.") 
After the presentation of the general scheme, participants, 
working in small groups, each with a leader from EPIC, and 
following the plan for developing objectives, actually 
practiced the writing of specific, measurable objectives in 
both cognitive and affective domains. Three small-group 
sessions were devoted to these activities; in the general 
evaluations made by individuals, these activities were cited 
as some of the most meaningful and practical experiences of 
the entire training program. 

Additional sessions were devoted to the techniques of 
constructing test items to measure the objectives identified 
in the planning stages of curriculum. The role of validity 
and reliability in measurement was emphasized. Samples of 
tables of specifications and item banks were introduced (See 
Appendix III) . 

Models of test items of various kinds were presented 
and discussed. Sample test questions measuring stated ob- 
jectives were prepared in practice sessions. Following the 
practice sessions in the writing of test questions and ex- 
amples of objectives, a session was devoted to the organi- 
zation of information. This included the use of coding 
systems and computers. Additional information regarding the 
analysis and reporting of information to decision makers and 
through the proper channels was supplied. 



Application to English Programs 



At the conclusion of the three and one half days of 
study of techniques of evaluation, the second phase of the 
training session began with the application of the model to 
the English discipline. Dr. Kellogg Hunt discussed research 
in how students write and presented specific techniques that 
could be applied to the evaluation of children's writing. 
These techniques included the use of the Cloze test, methods 
of sentence analysis by means of the t-unit and clause 
length, and imbedding or deep structure. In later sessions 
he demonstrated the application of systems of analysis based 
on syntax and transformational generative grammar, to facili 
tate objective measurements. Measurement of systactical 
growth in writing was illustrated. 

The second instructor in the direct application of 
specific objectives to the teaching and measurement of 
English curricula was Dr. Alan Purves. Using the teaching 
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of literature as a basis for his discussion, this consultant 
led the participants through the actual performance of read- 
ing poetry and a short story. At first, participants played 
the role of students and were instructed in analysis and in- 
terpretation. in later periods, participants devised teach- 
ing questions usable with literature. Evaluation techniques 
were applied to teaching literature in a practical presen- 
tation. A "grid" in indicated areas to be measured was com- 
parable to the EPIC model and might be used to focus on 
specific elements in defining further cube elements. 

Having developed skills in the construction of ob- 
jectives and evaluation instruments for measuring the ac- 
complishment of these objectives, and having covered evalu- 
ation in the teaching of composition, language, and 
literature, the participants were prepared for the next 
phase — the application of this knowledge to specific cur- 
ricula in specific schools. Since this group is expected to 
serve schools other than their own, it was logical that all 
of the skills learned at this training program needed to be 
combined into a system of procedures that might be applied 
to specific situations. The participants must not only be 
able to evaluate programs, but they must also know how to 
instruct others and aid in their evaluation. The appli- 
cation of the learnings was the subject of the portion of 
the program designed to get the participants started on 
their own. At this point, Mr. William Scannell helped to 
synthesize the heretofore disparate elements into a scheme 
for looking at specific English programs. Participants met 
together in small groups to establish procedures for imple- 
menting the program in their own geographic areas and to ar- 
range for cooperative work. 

One additional training session was conducted on the 
teaching of poetry, the writing of objectives to cover the 
teaching of poetry, and the development of instruments for 
evaluation of specific goals in the teaching of poetry. For 
this presentation. Dr. Stephen Dunning used his own vast 
knowledge and interest in poetry to spark the interest of 
participants in the application of the evaluative procedures 
previously experienced to a simulated teaching situation. 



Activities 

During the training session, many different activities 
were engaged in by the participants and the staff personnel. 
Most new information was presented in lecture to the total 
group with the aid of numerous transparencies, programmed 
books, charts, and duplicated materials. For intensive 
laboratory sessions on actual development of representative 
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materials for evaluation, small groups of 10-12 were used. 
Each group worked with a trained staff member. Some "home- 
work" assignments were produced as additional samples which 
were then evaluated by the staff. Many viewing sessions 
were held, some planned as part of the activities, some as 
an independent kind of sharing activity. Several partici- 
pants brought examples of teacher-made and student-made ma- 
terials? viewing areas were arranged for showing these ma- 
terials. Each evening the "MESSAGE Media Theatre" showed a 
variety of films which might be useful background for English 
teachers and curriculum leaders. The British Broadcasting 
Company films on improvised drama, for example, gave many 
participants a first opportunity to observe the British ap- 
proach to oral English. The films from the National Council 
of Teachers of English were useful both as an indication of 
the kinds of materials that might be obtained for local use 
and for the content of the films themselves. The humanities 
films, are, of course, familiar to many English teachers, but 
most participants had not had an opportunity to see all of 
them, especially the most recent ones (Appendix IV) . 

Committees were established early and meetings were 
held frequently in order to coordinate reporting to ad- 
ministrators and to the membership of the MCTE, to develop 
evaluation forms for the sessions themselves, and to plan 
procedures for follow-up. Every participant was engaged in 
some reporting presentation. Besides helping the individual 
to crystallize his thought and synthesize the various ex- 
periences of the training session into a meaningful whole, 
these reporting sessions also gave practice in making presen- 
tations to groups just as each will be doing as he helps to 
disseminate information. 



Availability of Resource Personnel 



One distinct advantage of the manner in which this 
session was conducted was the constant availability of the 
staff members, for both formal and informal contacts. Dr. 
Hammond, Dr. Armstrong, and the EPIC staff were available 
for three and one-half days; Dr. Hunt was available for 
three days. Dr. Purves for one day, and Dr. Dunning for two 
days. Mr. Scannell was present for the full training period, 
as were the MCTE personnel: Miss Mildred Webster, Miss 

Thelma McAndless, Mrs. Helen Wagner, and Dr. Taylor. Other 
MCTE executive board members were present for varying amounts 
of time. The availability of the staff and consultants gave 
many times for questions, discussion, and, in some cases, 
differences of opinion, and participants used their availa- 
bility during meals, over coffee, and in impromptu recre- 
ational activities. Another bonus of the living- learning 
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situation of the training sessions was the opportunity for 
constant interaction of participants* In their personal 
evaluations, most trainees mentioned the value of these con- 
tacts o 



Follow-up Activities 

Report to Administrators and to MCTE 

In setting the program Calendar for the MESSAGE train- 
ing program, the director included two sessions at which par- 
ticipants would report first to their administrators and 
second to the several hundred English teachers attending the 
Michigan Council of Teachers of English annual conference at 
Boyne Mountain, April 25-27* 

Both presentations were planned by committees of par- 
ticipants* They appeared before the two audiences (1) to 
share what they had been learning about effective ways to 
evaluate instructional programs, and (2) to show how 
knowledge, skills, and attitude in the behavioral domains 
could effect improvement in teaching the discipline of 
English * 

The audience of the administrative conference was com- 
posed of invited administrators, who had expressed an 
interest in having a staff member participate in the train- 
ing program, or their representatives* The responses of 
these administrators indicated that they had gained new in- 
sights into the need for a systematic approach to evaluation 
and the part that the writing of behavioral objectives may 
have in program evaluation* Several indicated that they 
planned to start work immediately with participants in the 
training program to set up local studies* 

The second general session of the annual meeting of the 
MCTE was devoted to the MESSAGE training program* In the 
presentation the committee used duplicate overhead projectors 
to make well-prepared and legible overlays visible in all 
parts of the large conference room* In the session, the 
main purpose of which was to generate enthusiasm for the 
project and give basic understanding to many people of the 
implications of long-range evaluation for English programs, 
the speakers introduced the EPIC model, gave a brief oppor- 
tunity for audience participation in building an objective 
that would incorporate the three dimensions plus that of 
measurement, and indicated ways in which a local school 
might begin to study the area of evaluation* 
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Evaluation 



The evaluation planned for the MESSAGE program is to be 
of two types: immediate and long-range. The immediate 

evaluation was planned to cover two types of responses. A 
pre- and post-test was, administered by the EPIC team over the 
objectives presented in the first part of the session. Re- 
sults , shown in Appendix V, are significant for each part of 
the instrument. 

The two Michigan Council of Teachers of English evalu- 
ators, Mrs. Helen Wagner and Miss Thelma McAndless, prepared 
and administered a questionnaire, asking for response in 
nine areas. The following quotations, chosen by the evalu- 
ators as presenting the complete range of responses to each 
section of the instrument, give an indication of the af- 
fective results of the session, as they are immediately ap- 
parent. (Procedures for gathering information about long- 
range results are discussed in the next segment.) 



The organization of the week-long session 

The careful planning of Project message was evident 
from the first encounter with the week-long program. Lo- 
cated in a spot conducive to quiet contemplation, the forty 
participants were introduced to a leisurely, yet firm de- 
velopment in the area of curriculum development, implemen- 
tation, and evaluation. It took about a day and a half for 
initial frustrations to wear themselves out of teachers who 
had left behind hectic schedules. But once they were re- 
laxed, it was evident that they were settled down to some 
real thinking. 

Advance knowledge of activities would have been helpful 
and would have enabled me to do some background work before 
coming . 

Variety in design and shifting focus on a range of con- 
cepts provided a tempo that established and maintained a 
sense of involvement. 

I think the week-long session has been well organized — 
not only have the speakers been informative, but there has 
been time to relax and assimilate the ideas presented. Also, 
there has been opportunity for professional reading — a real 
luxury! 

I 

4 

The Message Program provided for a good balance of time 
between large-group instruction and small-group instruction. 

* 

* 
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interest and 



The participants seem well chosen for ability, 
concern. 

Although the organization was apparent, a pre-conference 
agenda outlining the objectives of the conference, identify- 
ing topics to be covered, and a schedule of activities would 
have been most helpful. 

A most valuable part of the week has been the time left 
somewhat free for exchange of ideas among participants. So 
much has been given to us that we have needed time to chew . 
upon and digest, with others, the ideas we have dealt with. 



The quality and variety of consultative help available 



Completely satisfactory. 

Consultants and participants eagerly, helpfully, and 
willingly exchanged ideas all in the intere'st of and for the 
improvement of the evaluation process. This was a thrilling 
experience. To corner so much sincere dedication, enthusi- 
asm, and know-how is the epitome of achievement for Dr. 

Taylor and her committee . 

The consultants couldn't have been better and the re- 
source personnel have been most eager to listen and help. 

The consultants not only represented new and exciting 
ideas in a number of fields but also communicated their in- 
tent with clarity and warmth. 

EPIC section too much in too short a time — could have 
gone two more days. EPIC men very knowledgeable and helpful. 

In a few days time with national experts in research 
methods to guide us, I believe all of us here are now able 
to write our own program objectives with reasonably adequate 
measurement devices for reliability and validity, since the 
workshop required us to demonstrate this ability. 



The accessibility of resource personnel 

The EPIC team had a tendency to stick together. Live- 
in consultants prompted and stimulated further development. 
Ideas flourished at every dining table, lounge chair, and 
cottage room . 

All resource people were available; however, the use of 
these people might have been better planned. Evening 
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seminars could have been organized for the purpose of 
sharing such things as new programs , successful old pro- 
grams, techniques being implemented, and problems 
common to the participants involved in the project. A 
presentation and evaluation of the materials on display 
would have been more beneficial than just putting them 
on display. 

Everyone connected with project Message seemed 
anxious to offer his services and knowledge, and to my 
knowledge was ready and waiting to help. 

Good - consultants came here prepared to respond 
to questions and establish continuous communication 
with and among participants «. 

The interchange of ideas among participants 

Persons responsible for the selection of people for 
this group are to be complimented. It was a dynamic col- 
lection of individuals willing and eager to share their 
thoughts and experiences. 

There was a fair amount of interchange of ideas going 
on among participants, though after the first few days it 
seemed that many tried to avoid talking about curriculum 
during informal chats, perhaps because of "overexposure." 

It has been good, too, sharing at table, walking to- 
gether, gathering in small groups, with other people seeking 
some of the same answers you are, learning what has worked 
for them, contributing what you have done. The pace has 
been about right for absorbing without tiring. 

The interchange of ideas among participants has been 
the most valuable part of this program for me. I have never 
before known such an opportunity to learn about the teaching 
of English. 



The relevancy of content to needs of teachers and their 
institutions 

Some of the content was too time consuming for the busy 
classroom teachers. The Cloze test from Dr. Hunt's presen- 
tation seems just the quick test of readability that we 
need . 

This is one study program from which I shall return home 
confident I have some workable knowledge which will be 
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quietly earth-shaking but will not cause great tremors to 
the point of creating fear and hysteria. 

Whether we adapt, adopt or discard, MESSAGE has pro- 
vided some quide lines for the decision-making which must 
take place in curriculum development. More important, per- 
haps, than the specific messages is the attitude we've de- 
veloped: commitment to a specific program is possible. 

Perhaps the most important message was the writing of 
behavioral objectives which included the method of measure- 
ment. The T-unit and the Cloze test offer some immediate 
practical assessment of texts and student writing. 

Everything has been on a Paul Bunyan scale in both ex- 
cellence and quantity: the food, the scenery, the friendli- 

ness of participants and staff and the ideas presented. 

The relevancy of content proved itself against a high 
degree of skepticism beforehand. The leaders helped and al- 
lowed participants to make the content meaningful. 



The possibilities for implementation on local levels 



The principles of evaluation learned here will probably 
be implemented first in the classrooms of the participants 
then in other classrooms in the school, and on the other 
schools in the system. This should help to up-date English 
programs and, by means of proper evaluation, put them on a 
solid foundation. 

In implementing these evaluations personal appearances 
might be more dynamic than just printed material. How about 
appearances in English department meetings in the high 
schools in the county — as well as those in the city. How 
about going to junior high schools and elementary schools? 

This program will provide a solid beginning for the 
start of our evaluation of our new English program which 
goes into effect in September. I must say, though, that the 
follow-up procedures of measurement were barely touched upon 
in these sessions and many questions remain unanswered about 
testing and measuring effectively the objectives of our 
program. 

If a list of the participants was made available to 
each school along with a letter indicating that we are avail- 
able to discuss this program with any interested administra- 
tor or school staff* this study might have an impact on 
English in Michigan by helping schools to become more 
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efficient and effective in gauging the value and effect of 
the ir progr ams . 

Local implementation will be in direct proportion to 
the degree of influence (and naked power) the MESSAGE par- 
ticipant has within his own system. 

A close relationship should be established with each 
trainee by the director of the project, if this is done, 
there will be less opportunity for some of the members who 
are experiencing difficulty to slip off and become lost to 
the group. 

Dr. Taylor's suggestions of area grouping led us to dis- 
covery of method of implementation. We are now firmly set 
in a path which can be anything we want it to be if we work 
for it . 

Implementation may be hampered because teachers who 
have not had the experience of MESSAGE will not have a total 
commitment . 

Implementation is going to be difficult. I feel the 
only way to do this is to start small. There will not be 
the spread of MESSAGE that the directors hope for, but it 
will be forthcoming in time. 

The probable effects of MESSAGE upon the teaching of English 
in Michigan 

If behavioral objectives become a way of life in cur- 
riculum development, MESSAGE could become the miracle of the 
century . 

The probable effects should not be just limited to 
English but should involve all disciplines. If the EPIC 
principles are implemented, a revolution in the discipline 
(teaching) could take place. 

Long range effect will be felt in strengthened curricula 
and less frustration in the area of curricula development be- 
cause forty teachers know a little more about a lot of 
things . 

English teaching in the state can only profit from a 
project such as this. It will have made the teachers more 
aware of what others are doing, of what they might be doing, 
and, hopefully, of how they might go about effecting neces- 
sary changes in existing programs. 



V 



If we truly hope to affect state-wide English teaching, 
the next step may be intensive training of teams to work in 
different schools districts. Our MCTE regional districts 
might also be a possible organizational route. 

The effects of MESSAGE may not be immediately apparent, 
but I am sure that the impact will be felt in the schools 
represented and hopefully will be talked about enough so 
that other schools may seek similar help. Those people who 
work on North Central evaluating teams should be able to 
make other schools aware of MESSAGE concepts. 

Superior — has all kinds of possibilities. 

The effects of this conference upon the teaching of 
English in Michigan will be dependent upon the enthusiasm of 
members of this group and their ability to communicate with 
their administrators and their own local staffs, it seems 
evident that this kind of improvement is not for English 
alone but has a necessary relationship with every other 
discipline in the school . 

The effects lie in a well-organized follow-up program 
and interest and encouragement from local school districts. 

I believe MESSAGE will have an impact on the teaching 
of English in Michigan because: 

the participants will be better teachers, more aware of 
what they are doing anyway. 

the participants as a group in MCTE can help MCTE have 
a continuing impact. 

leadership and support from MCTE and NCTE can reinforce 
and extend the seeds planted here . 

Specific recommendations as to follow-up on this the first 
workshop of this type 

Have a second workshop to tabulate results in the field. 
Then have a third workshop for continued enlightenment. 

Have this group intercommunicate 7 perhaps work in teams 
of 2 or more to interpret MESSAGE to departments, admini- 
strators, perhaps school boards and community groups. 

Plan a follow-up meeting in each of several geographi- 
cal areas, possibly next fall to monitor action that has 
been an outgrowth of the conference. 

Have at least two follow-up surveys and a presentation 
at next years MCTE. 
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Perhaps some regional meetings could be scheduled to 
write out the nature of our successes as well as our ob- 
stacles or failures and what we consider to be the reasons 
for these various outcomes. The questionnaire which the 
follow-up group is sending out may give some hint as to how 
general our particular experiences have been and help us to 
draw up recommendations for future use. 

Realizing that one of the values of this conference has 
been the interchange of ideas, MESSAGE, or a like organi- 
zation, become an agency for gathering and disseminating of 
information regarding English programs within the state of 
Michigan. Funding for such an agency could be provided by 
local school districts making use of the services provided 
of a Title III project could be written. 

Plan a second workshop for the 1969-70 school year, 
implementing a team approach. Selected school districts 
would identify a team consisting of an English teacher, the 
department chairman, an administrator, and a testing and 
measurement person to work together in a pilot program for 
the development of an evaluation scheme. 

Follow-ups are difficult to maintain, but we must meet 
again next year to find out how we carried the MESSAGE. 



How can we encourage future sessions ? 

Apply for more federal money for follow-up workshops. 
Secure additional financial support in order to involve more 
teachers . 

Use literature sent to members of MCTE to supply infor- 
mation about these meetings at Boyne® 

Plan sessions to explore measurement - it was barely 
touched upon. 

Involve participants in planning for future sessions 
through development of surveys, questionnaires, inventories. 

Report this meeting in the Eng. Journal and other 
media. Let the school administration know who the speakers 
and consultants were and the implications of direct help be- 
ing given to those from their staff who attended. 

Improve communications concerning the activities via 
personal contact, publicity. Giant strides have obviously 
been made in the proper directions. 

I felt that for many, it was "too much, too fast," and 
that the relationships among the "Phases" of the conference 
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were not always clearly drawn. Plan for a slower pace for 
future sessions. 

The state department of education representative under- 
stands the project and its implications for the improvement 
of teaching of English in Michigan. Obtain money from that 
agency — by fall. Under the tension of producing, specific 
changes could come about. If future sessions are planned 
early enough, local areas could earmark some monies for such 
conferences . 

General 



This has been the most stimulating professional week I 
have ever spent. 

I am thankful for those who had the vision and the 
drive to organize this week's workshop, and for the oppor- 
tunity to be a participant. 



Long-range Evaluation 

Written into the MESSAGE Proposal were provisions for 
follow-up. At the Boyne Mountain training session the pro- 
ject participants attacked the ways and means for carrying 
out the proposals. Headed by Dr. Robert Fichtenau, Director 
of English, Intermediate District, Oakland Schools, Pontiac, 
a follow-up committee divided the areas of responsibility on 
a three-way basis: 

I Distribution and codifying of a 5-part survey designed by 
MESSAGE staff in consultation with the Division of Syste- 
matic Studies of Oakland Schools 

A. Purpose of Survey - Assessment of behavioral changes 
in MESSAGE Participants and schools represented 

B. Description of the Survey instrument (see Tentative 
Questionnaire, Appendix VII) 

1. Part one to find how differently the participants 
feel as a result of Project MESSAGE 

2. Part two to assess changes in behavior on the 
part of the individual participants 

3. Part three to gather information about specific 
changes in evaluation techniques being used in 
classrooms of the participants' schools and in 
schools in which the participants work 



22 



4 . 



Part four to survey attitudes in the participants' 
own schools and the administrators' attitudes to- 
ward utilizing the special knowledge and skills 
of the Project participant 

5. Part five to survey the extent to which infor- 
mation about Project MESSAGE has been disseminated 

C. Survey Schedule (September of 1969 and June 1970) 

D. Administrators' Questionnaire 

II Cooperation with the Michigan Department of Education 

(The following tentative commitment for assistance has 

been obtained from the Department of Education) 

A. Provide consultant service to those districts which 
wish to develop objectives and evaluation devices for 
their English programs 

B. Support additional workshops, on a limited scale , to 
enable English teachers to develop program objectives 
and evaluative techniques 

III Continuing Cooperation with MCTE 

A. At each of the Fall (1969) Regional Conferences of 

MCTE devote program time to "What's Happening in 
Region " as a result of Project MESSAGE 

B. At the annual spring meeting of MCTE (1970) schedule 
a session - Project MESSAGE - One Year Later. The 
committee planning this program will communicate with 
each of the MESSAGE participants on a scheduled basis 
throughout the school year. 
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APPENDIX I 
EPIC OBJECTIVES 
FOR 

PROJECT MESSAGE TRAINING SESSION 
Boyne Mountain Lodge 
April 18-21, 1969 



Participants attending the project Message/EPIC train- 
ing sessions* April 18-21, 1969, will: 

1. increase their knowledge of the EPIC Scheme for 
Evaluation as measured by their scores on the EPIC 
Feedback Questionnaire. 

2. increase their knowledge of program objectives 
terminology as measured by the EPIC Feedback 
Questionnaire and EIP tests. 

3. be able to apply their understanding of program ob- 
jectives, by writing an objective of their choice 
that includes the four EPIC criteria for objectives. 

4. will respond positively to the EPIC Training 
Sessions and staff as demonstrated by their re- 
sponses on the EPIC Feedback Questionnaire. 
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SKILL DEVELOPMENT IN EVALUATION FOR THE 
MICHIGAN COUNCIL OF TEACHERS OF ENGLISH 



EPIC Evaluation Center 
1034 East Adams 
Tucson, Arizona 



INTRODUCTION 



The nature of English programs will be quite varied 
from program to program; however, this does not alter the 
fact that a systematic scheme for evaluation is an absolute 
necessity. The instruments and techniques employed in the 
actual evaluation of each program may be different, but the 
step-by-step process should be consistent, or at least well- 
defined, for each program. This step-by-step process must 
be very understandable for personnel implementing English 
programs, particularly if such personnel have had limited 
training and experience in the area of evaluation. This 
paper presents a systematic approach to evaluation, the 
skills required of program personnel for its implementation, 
the technical assistance required to implement the system, 
and resources of the EPIC Evaluation Center available to pro- 
vide this training. 

I. Description of EPIC and its Funding Agency (Pima County 
Superintendents Office, 112 West Pennington, Tucson, 
Arizona) 

The EPIC Evaluation Center, established July 1, 1967, 
is supported by the U.S. Office of Education under the Ele- 
mentary and Secondary Education Act of 1965. The primary 
purpose of the Center has been that of developing and imple- 
menting a systematic scheme for the evaluation of instruction- 
al programs. Simultaneous with the development and implemen- 
tation of the EPIC approach, skill training workshops have 
been instituted by the Center as a part of its on-going 
respons ibilities . 

The EPIC Evaluation Center employs twenty-one staff 
members. These professionals each represent one or more 
specialties in the total evaluation process as described by 
EPIC. The unique characteristic of the EPIC Evaluation 
Center thus lies in its ability to facilitate the implemen- 
tation of the total evaluation process. 
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II . Evaluation Approach to be Presented in the Workshop 



The EPIC Structure and Scheme provide for the implemen- 
tation of procedures that lead to "objective evaluation." 
There are two essential differences between objective evalu- 
ation and the other less valid forms, such as subjective 
evaluation and external evaluation. 

The first difference is that of objectives. If a per- 
son is to make a valid decision based on the information 
collected in evaluation, he must know what he is evaluating. 
This is accomplished only when he is able to state his ob- 
jectives in behavioral terms. These objectives determine 
the procedures and criteria to be used in evaluation. The 
second difference is an outgrowth of the first. Objective 
evaluation guarantees that the people responsible for 
decision-making will become involved in the evaluation be- 
cause they are the ones who have to write the behavioral ob- 
jectives to be evaluated. 

The EPIC Structure of Variables Influe ncing In - 
structional Programs (Figure 1) facilitates the identifi- 
cation and description of the institutional and instruction- 
al variables affecting the program. Combining these vari- 
ables with the desired behavioral outcomes (cognitive, 
affective, or psychomotor) presents a simple, yet effective, 
method of developing program objectives. 

Another important feature of the EPIC Scheme for Evalu - 
ation (Figure 2) is that it facilitates not only "product" 
but also "process" evaluation. That is to say, consider- 
ation is given to the need for continuous feedback to enable 
program personnel to evaluate the progress of a program and 
make the necessary decisions to alter the program during its 
implementation, and not have to wait until the program is 
culminated . 

Basically, the procedures developed by EPIC demon- 
strate the philosophy that effective "objective evaluation" 
must be a product of the local district, agency, or project. 
Each school, agency, project, etc. must decide the nature of 
the objectives to be met. EPIC then provides the technical 
assistance, hardware, and in-service training programs be- 
yond the resources of the program to enable the personnel to 
implement evaluation. 

Ill . Evaluation of English Programs — General Comments 

The degree to which English programs meet their antici- 
pated outcomes must be determined through sound evaluation 
procedures implemented to measure specific objectives. This 
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EPIC SCHEME FOR EVALUATION OF INNOVATIONS 
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III . Evaluation of English Programs— General Comments 
(Continued) 

implies that prior to the initiation of each program, cer- 
tain steps in the evaluation process must be clearly defined. 
This would include adequate program description, objectives 
stated in behavioral terms, and an evaluation design which 
spells out the data to be collected, treatment descriptions, 
statistical analysis, and limitations of the evaluation pro- 
cedures. If we can accept the assumption that evaluation is 
simply a procedure for collecting pertinent information that 
can be used to facilitate decision-making, such objective 
evaluation seems to be valid. 



The evaluation procedure employed by each program must 
provide for the recognition of the many variables that may 
affect the given program, but most important, provision for 
the identification of outcomes in terms of cognitive, af- 
fective, and/or psychomotor behavior should be made — in other 
words, a means of identifying and classifying behavioral out- 
comes through the use of some organized system, such as the 
Taxonomy of Educational Objectives (Bloom) . In the training 
of personnel in the procedures of evaluation, it is impera- 
tive that: 



1. the personnel responsible for evaluation within 
each program be exposed to every phase of the 
evaluation process, and 

2. the personnel responsible for evaluation within 
each program develop the skills which they must 
ultimately employ in developing and implementing 
systematic evaluation. 

IV. The Evaluation Process and Related Skills 

The approach in providing experiences that not only ex- 
pose the participants to the evaluation process, but also in- 
creases their skills in handling various aspects of the 
evaluation process, should include the following: 

A. . Sequential Activities in B. Related Skill Development 
the Evaluation Process and Orientation, and EPIC 

~ Resources 



1 . Program Description 

Sound evaluation pro- 
cedures .require that the 
process begin with a de- 
scription of the planned 



1. Emphasis must be on 
the development of skills 
in identifying those vari- 
ables that are felt to be 
affecting that portion of 
the instructional program 
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IV. The Evaluation Process and Related Skills (Continued) 



program. The key to ef- 
fective evaluation is to 
identify and select what 
is to be evaluated. 



2. Goals and Objectives 

Evaluation as a pro- 
cess is best approached 
through objectives stated 
in behavioral terms . The 
objectives of a given pro- 
gram are developed follow- 
ing the identification of 
variables . 



under consideration. To 
develop these skills , a 
systematic, yet simple 
approach must be pro- 
vided to the program 
personnel • 

The EPIC Structure 
of Variables Influencing 
Instructional Programs 
is a three-dimensional 
structure which facili- 
tates the process of 
identifying and describ- 
ing the variables not 
only within the edu- 
cational setting, but al- 
so outside of it. The 
Structure considers the 
dimensions of instruction 
(organization, content, 
method, facilities, and 
cost) , institution (stu- 
dent, teacher, admini- 
strator, educational spe- 
cialist, family, and com- 
munity) and behavior 
(cognitive, affective, 
and psychomotor) . 

2. Skill sessions should 
be provided to program 
participants, in the. de- 
velopment and writing.i©f 
behavioral objectives. 

The staff of the 
EPIC Evaluation Center 
has available program ma- 
terials in writing be- 
havioral objectives. 

These materials have been 
especially designed to 
train educators who have 
had little or no experi- 
ence in this area. The 
EPIC Structure of Vari - 
ables Influencing In - 
structional Programs is 
utilized directly in 
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IV . The Evaluation Process and Related Skills (Continued) 



3. Development of Evalu - 
ation Designs and 
Functions of Data 
Processing 

Once the program ob- 
jectives have been stated, 
an evaluation design can 
be developed. This design 
is basically a description 
of how the data is to be 
collected and analyzed in 
order to determine if the 
objectives have been met. 



To facilitate the 
evaluation process, more 
often than not, program 
personnel must avail them- 
selves of data processing. 



teaching the skills of 
behavioral objective 
writing . 

3. Program personnel 
must have some orienta- 
tion to the consider- 
ations inherent in es- 
tablishing evaluation 
procedures. Assistance 
provided during training 
sessions should be di- 
rected toward each pro- 
gram. Participants 
should be exposed to such 
considerations as sample 
size, treatment descrip- 
tions, statistical an- 
alysis, and limitations, 
(note the words exposed 
and assistance ) Program 
personnel must be as- 
sisted in the development 
of "program calendars." 

A Program Calendar basi- 
cally consists of a 
schedule designating pre- 
cisely the evaluation 
procedures to be employed. 

The EPIC Evaluation 
Center maintains a com- 
plete Evaluation Division 
which has the primary 
functions of: (1) pro- 
viding technical as- 
sistance for the se- 
lection of treatments, 
measurement instruments, 
evaluation design, and 
statistical analyses, (2) 
processing data, and (3) 
reporting the results of 
the evaluation process. 

Program personnel 
must receive instruction 
in the data processing 
function in evaluation. 

The intent is not to 
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The Evaluation Process and Related Skills (Continued) 



4. Descript ion of Actual 

Program 

During the process of 
evaluation, an on-going de- 
scription of what is actual- 
ly happening in the program 
is a necessity. The actual 
program description should 
provide information as to 
whether the program is being 
carried out as planned. 



provide them with me- 
chanical skills related 
to data processing, but 
to provide them with a 
way of communicating 
with the evaluator and 
specific orientation in 
the most efficient 
methods of collecting and 
analyzing data. Exposure 
to such topics as the 
principal functions of 
data processing, storage 
of data, cost estimates, 
classifying data, and 
preparing data for com- 
puter processing are 
intended . 

The EPIC Evaluation 
Center has, as a part of 
its facility, the data 
processing equipment and 
technical personnel to 
describe and demonstrate 
the complete process of 
input, processing, and 
output . 

4. Program implementors 
need training and as- 
sistance in the points to 
be considered for con- 
tinual monitoring of the 
instructional program. 
Such skill would enable 
them to identify crucial 
points of reference in 
the program's progress 
and che identification 
of variations from the 
planned program. Such 
feedback would enable 
them to: (1) make the 
necessary changes in 
evaluative procedures 
(2) compare the evalu- 
ation results with spe- 
cific techniques of in- 
struction during the 
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XV . The Evaluation Process and Related Skills (Continued) 



5 . Output Analysis of 

Terminal Behavior 

Following the comple- 
tion of the program, the 
next step is that of 
analyzing the results of 
the evaluation to arrive 
at conclusions based on 
actual behavior. 



6 . Recycling Process 

With the program 
evaluated, , the school is u 
ready to consider changes 
in the program. The 
evaluator and the imple- 
mentor come together . De- 
cisions must be made rela- 
tive to program develop- 
ment • 



program, and (3) provide 
insights which could per- 
mit the incorporation of 
new content and procedure 
as the program progresses 

5. Program personnel 
must have' actual experi- 
ence in the analyzing of 
evaluation results as a 
part of their training. 
Such exposure would pro- 
vide them at least a 
limited skill in interpre 
tation of findings. 

The EPIC Evaluation 
Center has, as one of 
its major responsibili- 
ties, that of assisting 
program personnel in the 
interpretation of evalu- 
ation results. The pro- 
gram personnel are going 
to be making the decision 
to change the direction 
of their respective pro- 
grams; therefore, they 
must have the technical 
assistance in the in- 
terpretation of the re- 
sults of evaluation. 

6. Through the process 
of evaluation described, 
the recycling process en- 
tails the consideration 
of; (1) those objective? 
not met through actual 
program activities, (2) 
new or additional ob- 
jectives which might 
further improve the pro- 
gram, and (3) future 
evaluation procedures. 

The recycling process de- 
mands both the knowledge 
of the program on the 
part of the implemento?c 
and the technical skills 
of the evaluator. 



IV . The Evaluation Process and Related Skills (Continued) 



V . Workshop Leaders 



Dr. Robert J. Armstrong 
Assistant Professor 
College of Education 
The University of Arizona 
(Director, EPIC Evaluation 
Center) 

Dr. Robert E. Kraner 
Associate Professor 
College of Education 
The University of Arizona 
(Assistant Director, EPIC 
Evaluation Center) 



Dr. Terry D. Cornell 
Assistant Professor 
College of Education 
The University of 
Arizona 

(Coordinator, Evalu- 
ation Services EPIC 
Evaluation Center) 

Dr. E. Wayne Robertson 
Assistant Professor 
College of Education 
The University of 
Arizona 

(Coordinator, Field 
Services EPIC Evalu- 
ation Center) 



Special Consultants 



Dr. Robert L. Hammond 
Associate Professor 
School of Education 
Ohio State University 
(Ohio State Evaluation 
Center) 
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PROJECT MESSAGE 



Michigan Council of the Teachers of English 

WORKSHOP SCHEDULE 

(Participation by representatives of the 
EPIC Evaluation Center) 

FRIDAY, APRIL 18 3 1969 

Morning General Session #1 

"Introduction to the EPIC Evaluation Center and 

Workshop Expectations" 

Dr. Robert Armstrong 



1:30 - 3:00 p.m. 



Small Group Session #1 



"A Structure and Scheme for the Evaluation 
of Instructional Programs" 



3:00 - 3:20 p.m. 



Coffee Break 



3:30 - 4:30 p.m. Small Group Session #1 (Continued) 



"Developing Program Calendars and 
Program Monitoring Systems" 

SATURDAY, APRIL 19, 1969 



9:00 - 9:20 a.m. 



General Session #2 



"Introduction to Behavioral Objectives" 
Dr. E. Wayne Roberson 



9:30 - 10;15 a.m. 



Small Group Session #2 



"Developing and Writing Behavioral Objectives" 



10:15 - 10:35 a.m. 



Coffee Break 
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SATURDAY, APRIL 19, 1969 (Continued) 



10:35 - 11:45 a.m. (Small Group Session #2 (Continued) 



"Developing and Writing Behavioral Objectives" 




3:00 - 3:20 p.m. Coffee Break 

3:20 - 4:30 p.m. Small Group Session #3 (Continued) 

"Critiquing Behavioral Objectives" 

SUNDAY, APRIL 20 3 1969 

2:00 - 3:15 p.m. General Session #3 



"Measuring Cognitive Behavior" 
Dr. Terry D. Cornell 



3:15 - 3:30 


p.m . 


Coffee Break 




3:30 - 4:45 


p.m. 


General Session #3 (Continued) 



"Measuring Affective Behavior" 
Dr. Robert E. Kraner 
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MONDAY j APRIL 21, 1969 



9:00 - 


10:45 a.m. 


Small Group Session #4 




" Impleme nt a t io n 


of the EPIC Structure and 




Scheme 


for Evaluation" 


10:45 


- 11:00 a.m. 


Coffee Break 1 




11:00 


- 11:45 a-m. 


General Session #5 



"Pinal Summary Session" 
EPIC Staff 
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APPENDIX II 



PARTICIPANTS 
PROJECT MESSAGE 



Mr. Henry James Baron 
Calvin College 
1331 Franklin 
Grand Rapids, Mich. 49506 


Home: 1040 Alexander, S.E. 
Grand Rapids, Mich. 49507 
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Mrs. Mary Churchill 
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Mr. Neil B. Davis 
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Cheboygan Michigan 49721 


Rt. 3, Box 16 3A 
Cheboygan, Michigan 49721 
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St. Joseph Public Schools 
St. Joseph* Michigan 



Evaluator* Michigan Council 
of Teachers of English 



Executive Secretary* Michi- 
gan Council of Teachers of 
English 
Fiscal Officer 



43 



( 



I 

i 

I 



APPENDIX III 

SOME SUGGESTIONS FOR WRITING TRUE-FALSE ITEMS 

1. Base true or false items on propositions that are like- 
ly to be significant and useful in dealing with a wide 
variety of situations and problems. 

2. Express the proposition as simply and as clearly as 
possible. Use words whose meanings are definite and 
precise . 

3. Include enough background and information and qualifi- 
cations so that the truth or falsity of the statement 
does not depend on some special , uncommon assumption. 

4. Make the true statement true enough and the false state- 
ment false enough so that experts would unanimpusly 
agree on the answer. 

5. Choose and state propositions so that their truth or 
falsity is not immediately obvious to every reasonable 
person. 

6. Let each test item express a single idea. Avoid com- 
plex statements which unnecessarily combine several 
ideas. If it is necessary to include explanatory or 
qualifying elements in the statement, indicate clearly 
which part is to be judged true or false. 

7 . Do not create false statements from true ones simply by 
inserting the word "not." The task of judging the 
truth or falsity of a statement which says that some- 
thing is not so can become highly and needlessly 
complex . 

8. Word the statements so that sheer memory of words, 
empty phrases, or meaningless sentences will ,not permit 
a correct answer. 

9. Guard against irrelevant clues, such as specific de- 
terminers, which would permit a test-wise but unpre- 
pared examinee to respond correctly. Specific de- 
terminers • refer to sweeping terms like "all," "always," 
"none," "never," "impossible," "inevitable," etc. 
Statements including such terms are likely to be false. 
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SOME SUGGESTIONS FOR WRITING TRUE-FALSE ITEMS (Continued) 



On the other hand, statements including such qualifi- 
cations as "usually," "sometimes," "often," etc. are 
likely to be true . 



FAULTY TRUE-FALSE ITEMS 

1. Two pitfalls should be avoided in writing true-false 
items. (F) 

2. Camping has a good past, a better present, and an al- 
most unlimited future. (T) 

3. The relation between a parasite and its host is 
significant. (T) 

4. When you see a highway with a marker which says "Iowa 
U.S. 218" you know that the construction and upkeep of 
that road is built and maintained by the state and 
Federal governments. (T) 

5. Rain clouds are light in weight. (F) 

6. Merit is an important factor affecting a teacher's 
salary. (F) 

7. Frozen foods of the highest quality may be ruined in 
the kitchen. (T) 

8. Insurance agencies may be either specialized or 

general . (T) 

9. As a result of mistakes in our foreign policy, the Com- 
munists won control of China after World War II. (T) 

10. Life is a continuous process of choice-making, sacri- 

ficing one human value for another, which goes through 
the following steps: spontaneous mental selections re- 

garding everything we want, conflicting preferences 
hold each other in check, hesitation becomes deliber- 
ation as we weigh and compare values, finally a choice 
or preference emerges. (T) 

11. Columbus did not make four voyages of exploration to 
the Western hemisphere. (F) 

12. For every action there is an equal and opposite 
reaction. (T) 



FAULTY TRUE- FALSE ITEMS (Continued) 

13. Objective tests are useful in measuring all aspects of 
educational achievement. (F) 

14. Stars send out light that twinkles. (T) 



SUGGESTIONS FOR PREPARING GOOD MULTIPLE- CHOICE TEST ITEMS 

1. Develop multiple-choice test items on the basis of inde- 
pendently meaningful and demonstrably valid statements 
of relevant, important ideas. 

2. Choose item topics and ideas and write multiple-choice 
test items with a view to maximizing the discriminating 
power of the items . 

3. Arrange to write the initial item drafts so that subse- 
quent revision and assembly into the finished test will 
be as convenient as possible. 

4. Begin the item with a stem question or incomplete state- 
ment to which a reasonably adequate answer or completion 
can be given concisely and for which plausible wrong 
answers can be found. 

5. Phrase the intended correct response so that it is 
thoroughly correct or clearly adequate, expressed as 
clearly and concisely as possible, without providing 
clues which give it away to the clever but poorly pre- 
pared students. 

6. Choose and phrase the incorrect alternatives (dis- 
tractors) so that they are thoroughly wrong or clearly 
inadequate, yet plausible enough to appeal to sub- 
stantial-numbers of poorly prepared examinees. 

7. Word the item as clearly* simply, and correctly as 
possible . 

8. Modify the item, if necessary, so that it is reasonable 
to expect about half of the examinees to answer the 
item correctly. 

9. Arrange for competent, independent review and revision 
of the initial item draft. 



ILLUSTRATIONS OF MULTIPLE- CHOICE TEST ITEMS 

1 . Item permitting _a correct a nswer : 

What happened in 1953 to the proposed act of Congress 
granting statehood to Hawaii? 

a. It was passed by both the House and the Senate. 

b. It was defeated in both the House and the Senate. 



3 



47 



ILLUSTRATIONS OF MULTIPLE- CHOICE TEST ITEMS (Continued) 

*c. It was passed by the House but not by the Senate. 

d. No act regarding statehood for Hawaii was intro- 
duced in either the House or the Senate. 

2 . Item using citation of authority ; 

What does Ross say about the Kuder-Richardson method . 

for calculating test reliability? 

*a. It involves assumptions which are likely to be 

difficult to meet in the ordinary test situation. 

b. It produces coefficients which are higher than 
those obtained by the split-halves procedure. 

c. It is the simplest and generally most satisfactory 
method . 

d. It takes account of both pupil errors in response 
and sampling errors among the items . 

3 . Item requiring best answer : 

Which statement best characterizes the man appointed by 

President Eisenhower to be Chief Justice of the United 

States Supreme Court? 

a. An associate justice of the Supreme Court who had 
once been a professor of law at Harvard. 

*b . A successful governor who had been an unsuccessful 
candidate for the Republican presidential 
nomination. 

c. A well-known New York attorney who successfully 
prosecuted the leaders of the Communist Party in 
the United States. 

d. A Democratic senator from a Southern state who had 
supported Eisenhower's campaign for the presidency. 

4 . Item admitting no best answer : 

Which event in the following list has been of the 

greatest importance in American history? 

*a . Braddock's defeat 

b. Burr's conspiracy 

c. The Hayes-Tidden contest 

d. The Webster-Hayne debate 
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ILLUSTRATIONS OF MULTIPLE- CHOICE TEST ITEMS (Continued) 

5. Item dealing with an incidental detail : 

This question is based on the advertising campaign of 
, Maumkeag Mills to retain the market leadership of Pequot 

bed linen. What was the competitive position of Pequot 
products in 1927? 

a. Ahead of all competitors among all customers. 

*b. Strong with institutional buyers but weak with 
household consumers. 

c. Second only to Wamsutta among all customers, 
e. Weak with all groups of consumers. 

6 . Item based on unique organization of subject matter : 

The second principle of education is that the individual: 

a. Gathers knowledge. 

b. Makes mistakes. 

c. Responds to situations. 

*d. Resents domination. 

7 . Item requiring selective recall : 

Which of the following was an important development in 
Canada during 1953? 

*a. Rapid business and industrial growth. 

b. A severe and widespread economic depression. 

c. A marked trend toward Communism in provincial 
governments. 

d. Appearance of a strong movement favoring unifi- 
cation of the United States and Canada. 

8. Item using descriptive responses : 

What is monogamy? 

a. Refusal to marry. 

b. Marriage of one woman to more than one husband. 

c. Marriage^ of one man to more than onfe wife. 

*d . Marriage of one man to only one wife. 

9 . Item using label responses : 

A marriage in which one woman marries one man is called 

a. unicameral. 

b. dualism. 

t . c. monotheism. 

*d . monogamy. 
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ILLUSTRATIONS OF MULTIPLE- CHOICE TEST ITEMS (Continued) 

10. Item asking examinee his opinion : 

What do you consider the most important objective of 
the staff meetings? 

*a . to establish good working relations with your staff. 

b. to handle routine matters. 

c. to help teachers improve instruction. 

d. to practice and exemplify democracy in 
administration . 

11 . Item combining _a question with an explanation : 

Has the average size of farms in the United States 
tended to increase in recent years? Why? 

a. Yes, because as the soil loses its natural fer- 
tility more land must be cultivated to maintain the 
same output. 

b. Yes, because the use of farm machinery has made 
large farms more efficient than small farms. 

c. NOj because the difficulty in securing farm labor 
has forced many farmers to limit their operations. 

d. NOj because large family farms tend to be sub- 
divided to provide smaller farms for the children. 

12. Item using incomplete stem : 

Physiology teaches us that 

*a. the development of vital organs is dependent upon 
muscular activity. 

b. strength is independent of muscle size. 

c. the mind and body are not influenced by each other. 

d. work is not exercise. 

13. Item using negative stem : 

In the definition of a mineral which of the following 
is incorrect? 

a. It was produced by geologic processes. 

b. It has distinctive physical properties. 

c. It contains one or more elements. 

*d. Its chemical composition is variable. 
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ILLUSTRATIONS OP MULTIPLE- CHOICE TEST ITEMS (Continued) 
14 . Item asking no specific question ; 



In comparing the period of heterosexual adjustment of 
our culture with those of other cultures, it must be 
concluded that 

*a . there are tremendous differences that can only be 
explained on a cultural basis. 

b. there are large differences which must be explained 
by the interaction of biology and the more influen- 
tial culture. 

c. although there are some differences, the biological 
foundation of puberty is fundamental. 

d. in most cultures puberty is the period of hetero- 
sexual adjustment. 

15. Item including a necessary qualification : 

What change occurs in the composition of the air in a 
lighted airtight room in which the only living things 
are growing green plants? 

a. Carbon dioxide increases and oxygen decreases. 

*b. Carbon dioxide decreases and oxygen increases. 

c. Both carbon dioxide and oxygen increase. 

d. Both carbon dioxide and oxygen decrease. 

16 . Item involving an 11 instructional aside 11 ; 

In purifying water for a city water supply, one process 
is to have the impure water seep through layers of sand 
and fine and course gravel. Here many impurities are 
left behind. Below are four terms, one of which will 
describe this process better than the others. Select 
the correct one . 

a. Sedimentation 
*b. Filtration 

c. Chlorination 

d. Aeration 

17. Item showing nonparallel responses : 

Slavery was first started 

*a . at Jamestown settlement. 

b. at Plymouth settlement. 

c. at the settlement of Rhode Island. 

d. a decade before the Civil War. 
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ILLUSTRATIONS OF MULTIPLE- CHOICE TEST ITEMS (Continued) 

18 . Item including complicating elements in the responses : 

Systematic geography differs from regional geography 

mainly in that 

a. systematic geography deals, in the main, with physi- 
cal geography, while regional geography concerns it- 
self essentially with the field of human geography. 

b. systematic geography studies a region systematical- 
ly, while regional geography is concerned only with 
a descriptive account of a region, 

*c . systematic geography studies a single phenomenon in 
its distribution over the earth in order to supply 
generalizations for regional geography, which 
studies the arrangement of phenomena in one given 
area . 

d. systematic geography is the modern scientific way 
of studying differentiation of the earth's surface, 
while regional geography is the traditional and de- 
scriptive way of studying distribution of phenomena 
in space. 

19. Item using "none of these" appropriately : 

Which word is misspelled? 

a . Contrary 
*b. Tendancy 

c . Extreme 

d. Variable 

20. Item using stereotypes in distracters : 

Which of these has affected the greatest change in do- 
mestic plants and animals? 

a. Influence of environment on heredity. 

b. Organic evolution. 

c . Selective breeding. 

d. Survival of the fittest. 

2 1 . Item using obscure distracters : 

A chaotic condition: 

a. Asympotic 
*b. Confused 

c. Gauche 

d. Permutable 
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ILLUSTRATIONS OF MULTIPLE- CHOICE TEST ITEMS (Continued) 

22 . Item involving verbal trick : 

Horace Greeley is known for his 

a. advice to young men not to go west. 

b. discovery of anesthetics. 

*c. editorship of the New York Tribune . 
d. humorous anecdotes. 

23. Item showing ineffective expression : 

Among the factors listed which have contributed to 
rapid expansion of cut-over land, the most important 
during recent years has been 

a. confiscatory taxation of standing timber. 

b. high prices for lumber. 

c. rapid growth of population. 

d. rising standards of living. 

24. Items which interlock : 

What, if anything, developed in 1955 with respect to 
relations between Egypt and Soviet Russia? 

a. A Russian military force invaded Egypt. 

b. Egypt accused members of the Russian embassy of 
spying. 

c. Soviet Russia supported Egypt's demand for inde- 
pendence from Britain. 

*d. The Egyptian government arranged to buy arms for 
Soviet Russia. 

How did the Western powers react to Egypt's agreement 
to buy arms from the Soviet Russia? 

a. They approved it as a gesture of good will. 

b. The pointed out that it was a private affair be- 
tween Egypt and Russia. 

*c . They pointed out that it was a threat to peace in 
the Eastern Mediterranean. 

d. They protested that Egypt was aiding a potential 
e nemy . 
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ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued) 

25 . Item with response placed in tandem : 

The balance sheet report for the Ajax Canning Company 
would reveal 

a. the company's profit for the previous fiscal year. 
*b. the amount of money owed to its creditors. 

c. the amount of income tax paid. 

d. the amount of sales for the previous fiscal period. 

26 . Item with unnatural sequence of responses : 

The population of Denmark is about: 

a. 2 million 

to. 15 million 

*c. 4 million 

d. 7 million 
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SOME TYPES OF ESSAY QUESTIONS 



1 . Selective recall — basis given : 

Name three important developments in measurement that 
occurred during the first decade of the Twentieth 
Century. 

2 . Evaluation recall — basis given : 

Name three persons who have had the greatest influence 
on the development of intelligence testing. 

3 . Comparison of two things — cm ja single designated basis : 

Compare essay tests and objective tests from the stand- 
point of their effect on the study procedures used by 
the learner . 

4 . Comparison of two things — _in gene ral : 

Compare standardized and non- standardized tests. 

5 . Decision — for or against : 

In which, in your opinion, can you do better — oral or 
written examinations? Why? 

6 . Cause or effects : 

How do you account for the popularity of objective 
tests during the last fifty years? 

7 . Explanation of the use or exact meaning of some word , 
phrase , or statement in _a passage : 

What is the meaning of "objective" in the above 
question? 

8 . Summary of some unit of the text or of some article 
r ead : 

Summarize in not more than one page the advantages and 
limitations of essay tests. 

9 . Analysis ( The word itself seldom appears in the 
question ) : 

Why are many so-called "progressive educators" sus- 
picious of standardized tests? 
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SOME TYPES OF ESSAY QUESTIONS (Continued) 

10. Stateme nt of relationship.- 

Why is it that nearly all essay tests, regardless of 
the school subject, tend to a considerable extent to be 
measures of the learner's mastery of English? 

11 . Illustrations or examples ( the pupil 1 s own ) of princi - 
ples in science , construction in language , etc . 

12 . Classification : 

What type of error appears in the following test item? 
(With what country did the United States fight during 
World War II?) 

13. Application of rules , laws , or principles to new 
situations : 

In the light of experience in the United States with 
examinations for selecting college students, what 
public- relations problems would you expect to arise in 
England because of the Age 11+ examinations there? 

14 . Discussion : 

Discuss the role of Sir Francis Galton in the develop- 
ment of the Pearson Product-moment coefficient of corre- 
lation, _r. 

15 . Stateme nt of aim — author 1 s purpose in his selection or 
organization of material : 

Why are individual mental tests not treated in greater 
detail in this book? 

16 . Criticism — ^s jto the adeguacy , correctness , or relevancy 
of _a printed statement , or _a classmate 1 s answer to ja 
Question on the lesson : 

Criticize or defend the statement, "The essay test over- 
rates the importance of knowing how to say a thing and 
underrates the importance of having something to say." 
"To criticize" assumes a set of standards given or 
known. Many persons are under the impression that it 
means "Tell something good about and tell something bad 
about, but mostly good, so that it will be construc- 
tive." Teachers should not foster this misconception. 
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SOME TYPES OF ESSAY QUESTIONS (Continued) 

17 . Outline ; 

Outline the principal steps in the construction of an 
informal teacher-made test. 

18 . Reorganization of facts (a good type of review guestion 
to give training in organization ) : 

Name ten practical suggestions from this book that are 
particularly applicable to the subject you teach or 
plan to teach. 

19 . Formulation of new questions — problems and questions 
raised ; 

What are some problems relating to the use of essay 
tests that require further study? 

20 . New methods of procedure : 

Suggest a plan for proving the truth or falsity of the 
contention that exemption from semester examinations 
for the ablest students is a good policy in high school. 

21. Inferential thinking ; 

Is the author of this book likely to use essay tests 
frequently in his measurement classes? 
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PROGRAM CALENDARS 



A Program Calendar should contain the following informations 

1. Objectives — the objectives of the given program should 
be stated in behavioral terms, along with the specifi- 
cation of how these objectives are to be measured. 

2 . Program Description — the program description should be a 
detailed description of what is going to take place in 
the program. Each variable along the instructional and 
institutional dimensions should be considered and those 
variables which are going to be directly affecting the 
given program should be expanded upon in the program 
calendar. For example, each variable along the insti- 
tutional axis should be considered when the sample is de- 
scribed. Is the sample going to be of students, 
teachers, administrators, etc., or a combination of 
these? The size of the sample, grade level, ethnic back- 
ground, and other pertinent data should be included when 
the description of the treatment is considered. What 
content and materials are going to be used? What 
method (s) is/are going to be implemented? How about fa- 
cilities? Is there going to be a change in organi- 
zation? Next, an evaluation design should be specified. 
This could be a picture showing data collection points 
and the comparisons to be made. In addition, the sta- 
tistical analysis should be described. 

3. Monitoring System — included in the program calendar 
should be a description of the monitoring system that is 
going to be used to check to see if the planned program 
went as planned. An example of a format which could be 
used is attached. Notice that the monitoring system de- 
scribes, by objective, what is going to take place. 

Then, when the actual program is implemented, it is a 
matter of comparing what was planned to what actually 
took place by using the same format. 

Calendar of Events — a calendar of events should be drawn 
up to describe the sequence of events in terms of data 
collection and other important responsibilities. 
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PROGRAM: 



Involvement o£ Teacher Effectiveness Through In-Service Training 



FORMAT FOR MONITORING 



Monitoring 



Period 


Related Program Objectives 


Organization 


Example 


Objective #1: Teachers par- 


Time: Two one- 




ticipating in the in-service 


hour sessions 


Week of 


program will respond with 




September 


increased flexibility as 


Space: Small 


18-22 


measured by coded video 


groups, each in- 




tapes of their classroom 


volving four 




teaching utilizing coding 


teachers and one 




systems of Interaction 


resource person. 




Analysis and Teacher 


Teachers random- 




Self-Appraisal . 


ly selected for 
groups. 



Week of 


Objective #1: 


Time: One -hour 


September 


Same as above. 


session 


25-29 




Space: Large 
group structure, 
with no group- 
ing implied. All 
participants in 
large groups. 



Description of Instructional Variable 



Content 


Me thod 


Facilities 


Cost 


Teaching Skills 
Designed to im- 
prove classroom 
interaction . 
Basic elements 
of: 


Teaching Activities 
Leader acting only 
as a resource per- 
son . 

Types of Interaction 


Equipment 
and Related 
Activities 
Video Taping 
of teachers 
in their re- 


Not 

Appli- 

cable 


(1) listening, 

(2) questioning, 

(3) utilization 
of student 
ideas , 

(4) structuring 
responses 


Participants in each 
group will assume a 
pre -determined role 
for each exercise. 


spective 
classrooms 
(one -ha If 
hour per 
teacher par- 
ticipant) . 




Use of programmed 
materials 









Skill development 


Teaching Activities 


Equipment 


Not 


in the coding of 


Lecture and discus- 


and Related 


Appli- 


teacher-pupil 
verbal interaction 


sion method 


Activities 
Video tape 


cable 


using Flanders* 


Types of Inter- 


equipment 




Interaction Analy- 


action 


and demon- 




sis System. 


Principally resource 
per son- -teacher 
participants . 


stration 
tapes . 





*Time in the organization variable refers to the duration and/or sequence of 
blocks of time devoted to the subject matter. 

*Space refers to the general structure of the group to classify the parti- 
cipants . 



APPENDIX IV 



MESSAGE MEDIA THEATRE 



Friday Evening , April 18 

BBC Film: Improvised Drama I 

The Meaning of Modern Painting 
Huckleberry Finn I 

Saturday Evening 

BBC Film: Improvised Drama II 

The Odyssey 

The Themes of Macbeth 

Sunday Evening 

The Magic Prison (Emily Dickinson) 

\ The Age of Elizabeth 

The Poisoned Kingdom (Hamlet) 
i Hamlet II 

Hamlet III 

Monday Evening 

MCTE Elementary English Series: Word Geography 

Art of Huckleberry Finn 

Huckleberry Finn and the American Experiment 
f (check TV for the Spoon River Anthology) 

Tuesday Evening 

Perspectives of the Imagination: Comparisons (NCTE) 

Toronto Board of Education, classroom movies 

Wednesday Evening 

Marshall McLuhan I and II 

(poetry reading session, participants) 

Thursday Evening 

The Theatre — One of the Humanities 
Our Town I 

: Our Town II 

I 

t 

* 

* 

i 

| 

i 

i 

* 

t i 

| 
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APPENDIX V 



EPIC FEEDBACK QUESTIONNAIRE 
PROJECT MESSAGE TRAINING SESSION 
Boyne Mountain Lodge 
April 18-21, 1969 



SCHEME FOR EVALUATI ON 

Place the letter in the correct response in each blank. 



1. 


The 


EPIC Evaluation 


Center 


began 


operation in 




a • 


1961 




c . 


1967 




b. 


1966 




d . 


1964 


2. 


The 


EPIC Evaluation 


Center 


is located in 




a • 


Phoenix 




c . 


Washington 




b. 


De nve r 




d . 


Tucson 


3. 


The 


EPIC Evaluation 


Center 


is funded through 




a • 


ESEA Title I 




c . 


NTL 




b. 


NDEA Title III 




d . 


ESEA Title III 


4. 


The 


EPIC Evaluation 


Center 


has two divisions 




a . 


Field Services 


and Computer 


Services 




b. 


Public Relations and Evaluation 




c . 


Field Services 


and Evaluation 




a. 


Measurement and 


Evaluation 




5. 


Phase three of the 


Scheme 


for Evaluation is 




a • 


feedback • 




c . 


measurement . 




b. 


planning . 




d . 


implementation . 


6. 


The 


Program Calendar is part of 


the Phase . 




a . 


Feedback 




c . 


Measurement 




b. 


Planning 




d. 


Impleme ntation 
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PROJECT MESSAGE TRAINING SESSION 
April 18-21, 1969 



SCHEME FOR EVALUATION (Continued) 



7. 



8 . 



The Recycling 

a . Feedback 

b. Planning 

Monitor lag is 



Process is part 



part of the 



of the Phase . 

■ - r — '^ r -- 

c. Measurement 

d. Implementation 

Phase . 



a. Feedback 

b. Planning 



c. Measurement 

d. Implementation 



9. A Program Calendar does not contain . 

a. objectives. c. recycling procedures, 

b* data to be gathered. d. feedback dates. 



10 o The EPIC Structure of Variables Influencing In- 
structional Programs does not contain a(n) 

Dimension. 

a. Behavior c. Institution 

b. Measurement d. Instruction 



OBJECTIVES 

1. There are dimensions in the EPIC Structure of 

Variables Influencing Instructional Programs. 

a. three c. five 

b. four d. two 

2. Utilizing the EPIC Structure of Variables Influencing 

Instructional Programs, variables form a factor. 

a • two c . six 

b. four d. three 

3. is a variable of the Instructional Dimension. 



a* Administrator 
b. Facility 



c. Student 

d . Knowledge 
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me FEEDBACK UUKtfHONtfAtKl'i 
PROJECT MESSAGE TRAINS SESSION 
April 18-21 | 1988 



OBJECTIVES (Continued) 

4. factors can be formed utilizing the EPIC Structure 

of Variables influencing instructional Programs. 

a. 10 c. 325 

b. 150 d. 420 



5 . Team teaching would be an 
al variable? 



a* Content 

b. Facility 

Match the following: 

6 . Application 

7. Respond 

8. Analysis 

9 . Energy 

10. Receive 

11. Knowledge 

12. Comprehension 

13. In writing a program obj 
proach, the following in 

a* Behavior variable, 

b. How behavior will be 
measured. 



example of which Instructton- 



c . Method 

d. Organization 



A. Affective 



C. Cognitive 



P. Psychomotor 

ive, utilizing the EPIC ap- 
mation is unnecessary . . 

c. Minimum acceptable 
performance 

d. Instructional variable. 



14. Write a program objective of your choice in the space 
below, underline the four EPIC criteria. 
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EPIC FEEDBACK QUESTIONNAIRE 
PROJECT MESSAGE TRAINING. SESSION 
April 18-21, 1969 

MEASUREMENT 

1. Validity refers to 

a. how much a test costs in relation to the acquired 
information. 

b. the degree a test measures what it purports to 
measure . 

c. the degree of difficulty in interpreting test 
scores . 

d. the number of equivalent forms accompanying the 
test . 

2. When one is concerned with reliability, he should look 

at the . 

a. ease of interpretation of test scores. 

b. number of uses for the test. 

c. cost of the test. 

d. consistency of test scores. 

3. The most common level of cognitive behavior measured by 

a standardized test item is . 

a. comprehension. c. knowledge. 

b. analysis. d. application. 

4. The relationship of validity and reliability is . 

a. There is no relationship. 

b. Reliability is not a necessary, but a significant, 
condition for validity. 

c. Reliability is a necessary and sufficient condition 
for validity. 

d. Reliability is a necessary but not a sufficient 
condition for validity. 

5. When are the measuring instruments to be used in evalu- 
ation specified? 

a. When the behavioral objectives are written. 

b. When the monitoring system is developed. 

c. During the output-analysis step. 

d. During the recycling phase. 
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EPIC FEEDBACK QUESTIONNAIRE 
PROJECT MESSAGE TRAINING SESSION 
April 18-21, 1969 

MEASUREMENT (Continued) 

6. The highest level of measurement of Affective Behavior 

in the classroom is usually on the level. 

a. receiving c. valuing 

b. responding d. characterization 

7. A table of specifications will help the teacher to de- 
termine the . 

a. amount of achievement. c. validity of the test. 

b. type of test items being d. standard divisions of 

used. the measuring 

instrument . 

8. Which two dimensions from the EPIC Structure are uti- 
lized in developing a Table of Specifications for 
measurement in the affective area? 

a. Behavior — Instruction c. Institution — Behavior 

b. Instruction — Institution d. Instruction — Measure- 

» me nt 

;* Circle One: SD = Strongly Disagree D = Disagree 

Undecided = U A = Agree 

SA = Strongly Agree Percentages 

Omit 

1. As a result of this workshop, 



; 




I have more knowledge of 


0 


0 


5 


51 


41 


2 






evaluation. 


SD 


D 


U 


A 


SA 




F- 

f ' 


2. 


When I hear the word "ob- 














i 

! 




jectives," I have a feeling 


39 


46 


10 


2 


2 


0 


t 

r 




of dislike. 


SD 


D 


U 


A 


SA 




l 


3. 


Members of the workshop 
staff really knew their 


0 


0 


5 


41 


54 


0 


? 




subject . 


SD 


D 


U 


A 


SA 






4. 


There was too much lecture 


10 


54 


17 


12 


5 


2 






and too little interaction. 


SD 


D 


U 


A 


SA 






5. 


Competence in evaluation 


















will facilitate teaching 


2 


0 


5 


37 


56 


0 






procedures . 


SD 


D 


U 


A 


SA 








f. 

© 
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EPIC FEEDBACK QUESTIONNAIRE 
PROJECT MESSAGE TRAINING SESSION 
April 18-21, 1969 



MEASUREMENT (Continued) 

















Omit 


6. 


Writing objectives is neces- 


2 


5 


7 


37 


49 


0 




sary for effective teaching. 


SD 


D 


U 


A 


SA 




7. 


As a result of this work- 
shop, I have more knowledge 


0 


0 


5 


41 


54 


0 




of objective writing. 


SD 


D 


U 


A 


SA 




8. 


As a result of this work- 
















shop, I will be able to use 
teacher-made measuring 


0 


7 


10 


44 


39 


0 




instruments more effectively. 


SD 


D 


U 


A 


SA 




9. 


The large group meetings in 
this workshop were interest- 


2 


2 


7 


71 


17 


0 




ing and helpful . 


SD 


D 


U 


A 


SA 




10. 


The small group meetings in 
this workshop were interest- 


0 


2 


2 


32 


61 


2 




ing and helpful. 


SD 


D 


U 


A 


SA 




11. 


Generally speaking, this 
workshop provided me with 


5 


2 


15 


39 


37 


2 




what I came for . 


SD 


D 


U 


A 


SA 
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EPIC FEEDBACK QUESTIONNAIRE 



PROJECT MESSAGE TRAINING SESSION 
Boyne Mountain Lodge 
April 18-21, 1969 



i 

i 

[ 



t 



TOTAL TEST RESULTS 
Pre-Test Mean 
12.95 

SUB- PART RESULTS 

1 . Scheme for Evaluation 
Pre-Test 

4.25 

2 . Objectives 
Pre-Test 

4.90 

3 . Measurement 
Pre-Test 

4.03 



Post-Test Mean 
22.35 



Post-Test 

6.55 

Post-Test 

10.64 

Post-Test 

5.36 
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t value 

10.00 



t value 
6.57 

t value 
10.07 

t value 
4.59 



APPENDIX VI 



NAME (Optional) 

EVALUATION OF BEHAVIORAL OBJECTIVES OF MESSAGE 



In compiling a report for the United States Office of Edu- 
cation, the staff needs a statement about your reactions to 
MESSAGE. GIVE YOUR EVALUATION TO HELEN WAGNER NOT LATER 
THAN LUNCH TIME ON WEDNESDAY. Thanks. 



I. Please submit % to 1 page covering the following items: 

1) the organization of the week-long session 

2) the quality and variety of consultative help 
available 

3) the accessibility of resource personnel 

4) the interchange of ideas among participants 

5) the relevancy of content to needs of teachers and 
their institutions 

6) the possibilities for implementation on local 
levels 

7) the probable effects of MESSAGE upon the teaching 
of English in Michigan 
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II. Please list your specific recommendations as to follow- 
up on this the first workshop of this type . Do you 
think we should encourage future sessions of this type? 




How can we encourage future sessions? (Be specific) 
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APPENDIX VII 



TENTATIVE FOLLOW UP EVALUATION QUESTIONS 

Please add suggestions and return Thrusday morning to Bob 
Fichtenau 

1. Have you presented EPIC behavioral objectives to your 
staff? 

2. Are you going on alone? 

3. What measurement devices did you use, if any? 

4. Which institutional variables did you write objectives 
for? 

5. In your school, have you written objectives for: 

a. cognitive behavior 

b. effective behavior 

c. psychomotor behavior 

6. Did you write objectives stressing - organization, con 
tent, etc.? 

7. Have you introduced the EPIC concept within 

your building 
English staff 
other teachers 

8. Have you introduced the EPIC concept at the administra 
tive level? 

9. Did you talk about the EPIC concept beyond 

your department 

your school 

your school district 

10. Did you use a standardized test? 



a . 

b. 

c . 
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11 . 



Did 



you use teacher made tests? 



a . 

b . 



12. Did this test do what you wanted it to do? 



13. What type of curriculum does your school have? 

a • 

b. 

14. What kind of program do you have in your school? 

15. Would you be willing to share it? 

16. What are you doing about writing behavioral objectives? 

17. If responses are negative * why are they negative? 

18. From whom did you get the most support? 

19. Was in-service training involved in the introduction of 
this program? 

20. Has there been any parent involvement? 

21. Did any of the institutional variables aid you in 
writing cognitive objectives? 

22. Outside your own district personnel, from whom have you 
received help in utilizing your Project MESSAGE 
training? 

a. other MESSAGE participants 

b. Intermediate district 

c. State Dept, of Ed. 

d. University 

e. Epic Evaluation Center 

23. As a result of your training, are changes taking place 
in your school system? 

a . no change 

b. program changes are in the planning stage 

c. objectives are being developed 

d. both b & c 

e. b & c in addition to developed evaluation 
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24. Prom whom have you received the most encouragement to 

put to use the skills you developed by participating in 
MESSAGE? 

a. your own department 

b. your principal 

c. curriculum director or specialists 

d. superintendent 

e. no support 
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